Repository navigation
fix(ocr): recover from Vision failures on macOS 27 - #95
Merged
Merged
Conversation
Vision's accurate recognition path can fail or stall at the Neural Engine layer on macOS 27 (E5RT error code 13), leaving captures failing silently and the app unresponsive until restart. - Retry with a fresh request at the fast recognition level when handler.perform throws at the accurate level - Bound each Vision attempt with a timeout so a stalled model compilation cannot freeze the capture pipeline - Fall back to a configured LLM or Tesseract engine when Vision still fails, skipping an engine that already failed - Surface a user-facing notification (stderr for the CLI) instead of failing silently; watch mode logs instead to avoid spam
This was referenced Sep 26, 2026
The legacy VNRecognizeTextRequest revision-3 path is broken OS-side on macOS 27 (issue #92): after a process's first successful request, every subsequent request fails with E5RT error code 13 until restart. The modern Swift Vision API is unaffected, so route VisionOCREngine through RecognizeTextRequest wherever it exists, mapping recognition level, languages, correction, and custom words onto the new request and keeping the fast-level retry. The legacy path remains for macOS 14.x, and the engine-fallback safety net is unchanged.
Review findings on #95: the Vision-failure fallback passed Vision's language list to Tesseract (empty under automatic detection, so Tesseract defaulted to English); it now uses the configured Tesseract languages. The accurate-level retry no longer fires on CancellationError from an abandoned timed-out task. Recognition tests now assert the accurate level actually served the result so the silent fast retry cannot mask the issue #92 regression.
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #92
Root cause
On macOS 27, Vision's
.accuraterecognition path runs a model through the Neural Engine compiler (E5RT). When compilation fails,handler.performthrows (E5RT Error ... Code: 13);performVisionOCRcaught the error, logged it, and returned nil — the capture failed with no feedback.Reproduced part of this locally on macOS 27.0 (26A428): the first
.accuraterequest in a fresh process stalled for 30.5s while the system compiled the model, then succeeded (cached afterward). That matches the reported 10+ second lockup:performVisionOCRhad no timeout, soisCaptureInProgressstayed set for the whole stall and every menu action was silently ignored (beginCaptureTransactionguard). The hard E5RT failure did not reproduce here (compilation succeeds on this machine), but.fastcompleted in 0.01–0.03s in all runs — it uses a different model path, which is why it's the retry level.Migration to the modern Vision API (macOS 15+)
@hiyotim's testing in #92 pinned down the failure mode precisely: with the legacy
VNRecognizeTextRequest(revision 3), the first request in a process succeeds (~15s), then every subsequent request fails in ~2ms with E5RT Code 13 until the process restarts — reproduced outside TRex with a plain request loop.VNRecognizeTextRequestRevision2works but lacks Cyrillic; the modern Swift Vision API (RecognizeTextRequest) works consistently, including Russian. The same regression is reported in m-tkg/clipkun#12/#14 and jfarcand/mirroir-mcp#36.So
VisionOCREnginenow routes through the modernRecognizeTextRequestAPI, gated withif #available(macOS 15.0, *):.accurate/.fast),recognitionLanguages(BCP-47 strings converted toLocale.Language),usesLanguageCorrection, automatic language detection, custom words, minimum text height — and identicalOCRResultconstruction (joined top candidates, averaged confidence, level string)..accurate, retry once at.fastwith a fresh request.VNRecognizeTextRequestpath (with its.fastretry) remains for macOS 14.x, and everything below stays as the safety net for any remaining failure.VNRecognizedTextObservationhandling is internal to the engine, and table detection uses the separateRecognizeDocumentsRequestpath (already the modern API).Verified against the reported failure mode on this macOS 27 machine: a new regression test runs 6 sequential
.accuraterecognitions on rendered images through the migrated path — all succeed with correct text (the legacy loop's failure begins at request 2, per hiyotim's reproduction). A second new test confirms Cyrillic recognition, the capability that ruled out the revision-2 workaround.Changes
VisionOCREngine.recognizeText: on macOS 15+, use the modernRecognizeTextRequestAPI (immune to the E5RT regression); on macOS 14.x, keep the legacy request. Either way, when recognition throws at.accurate, retry once with a fresh request at.fastbefore propagating the error. Covers every Vision caller (capture, clipboard, watch mode, automation, LLM fallback).performVisionOCR: each attempt is now bounded by a 10s timeout (same pattern asperformOCR's existing 5s timeout), with a.fastretry on timeout or error. A stalled model compile can no longer freeze the pipeline; the in-progress flag always resets (alreadydefer-guarded on all paths, now bounded too).recognizeImagerouting.Verification
swift test --package-path Packages/TRexCore: 30 tests pass, including 9 covering fallback engine selection, fallback execution with injected failing/fixed-resultOCREnginetest doubles, both recognition levels on a rendered image, and in-progress state reset after a failed capture — plus 2 new ones: repeated sequential recognition through the modern path (issue macOS 27 issues #92's exact failure mode) and Cyrillic recognition.xcodebuild -scheme TRex -configuration Debug build CODE_SIGNING_ALLOWED=NO -skipMacroValidation: builds clean.